An Investigation of Dirichlet Prior Smoothing’s Performance Advantage

In the language modeling approach to information retrieval, Dirichlet prior smoothing frequently outperforms Jelinek-Mercer smoothing. Both Dirichlet prior and Jelinek-Mercer are forms of linear interpolated smoothing. The only difference between them is that Dirichlet prior determines the amount of smoothing based on a document’s length. Our hypothesis was that Dirichlet prior’s performance advantage comes from an implicit document prior that favors longer documents. We tested our hypothesis by first calculating a prior for a given document length from the known relevant documents. We then determined the performance of each smoothing method with and without the document prior. We discovered that when given the document prior, Jelinek-Mercer smoothing matches or exceeds the performance of Dirichlet prior smoothing. Dirichlet prior smoothing’s performance advantage appears to come more from an implicit prior favoring longer documents than from better estimation of the document model.